On Supervised On-Line Rolling-Horizon Control for Infinite-Horizon Discounted Markov Decision Processes

This note re-visits the rolling-horizon control approach to problem of Markov decision process (MDP) with infinite-horizon discounted expected reward criterion. Distinguished from classical value-iteration approaches, we develop an asynchronous on-line algorithm based on policy iteration integrated a multi-policy improvement method switching. A sequence monotonically improving solutions forecast-horizon sub-MDP is generated by updating current solution only at currently visited state, building in effect for MDP over infinite horizon. Feedbacks “supervisors,” if available, can be also incorporated while updating. We focus convergence issue relation transition structure MDP. Either global optimal or local “locally-optimal” fixed-policy finite time achieved depending structure.